Analyzing Data Management Plans: Where Librarians Can Make a Differ- ence

Since January 18, 2011, any researcher applying to a National Science Foundation (NSF) grant must include a data management plan (DMP) in their proposal. Many librarians have responded to this mandate by establishing new data­related services. One potential area for engaging with researchers in the grant proposal process is offering a DMP review service. In preparation for offering a DMP review service, engineering librarians at the University of Michigan (U­M) reviewed twenty­nine DMPs that were part of successful NSF grant proposals accepted in early 2014. The librarians analyzed the DMPs using three different sets of criteria: two rubrics developed at U­M, and a rubric currently under development by the IMLS funded Data management plan as a Research Tool (DART) project. The librarians had access to this third set of criteria as its trial run. Analysis of the DMPs shows that the overall quality of DMPs at U­M varies greatly. Some common weaknesses in the DMPs are: lack of roles and responsibilities; lack of metadata standards that will be used; and failure to mention intellectual property rights. Analysis of the DMPs also revealed gaps in the librarians’ knowledge of DMP requirements. In addition to discussing the findings from this current set of analyses, overall DMP quality from this study is compared to DMP quality found in a similar analysis of engineering DMPs from 2013. Looking toward a future where the outcome of grant proposals may be more dependent on the quality of the DMP, this analysis gives the engineering librarians at U­M a foundation for creating a DMP service in the coming year, and can inform other librarians who wish to develop a similar service at their institution. Introduction In 2011, the National Science Foundation (NSF) began requiring researchers to include a data management plan (DMP) as a part of their submitted proposals for funding.​1​ As defined by the NSF, a DMP should include: ● a description of the data being developed, ● the standards that will be employed in formatting and developing the content of the data and metadata produced, ● policies for accessing and sharing the data with others, ● statements on how the data may be re­used, re­distributed or used to produce derivatives, and ● how the data will be archived to ensure access in the long­term.​2 Some of the directorates within the NSF have produced additional requirements or gone into more detail about what they expect to see in DMPs. The NSF Directorate for Engineering, for example, states that a DMP should “outline the rights and obligations of all parties as to their roles and responsibilities in the managing and retention of research data.”​3 Even before the requirements were enacted, librarians were discussing the impact of data management plans and how libraries could respond through offering consultation and other services.​4​ Librarians recognized early on that developing and implementing DMPs would likely prove to be a challenge for most researchers, an assumption confirmed by the Cornell University Library in their investigation of faculty reactions to these requirements in 2012.​5​ Delserone notes that “there is growing recognition, within the library profession and the scientific community as well, that our combined knowledge and skills may be valuable to this [data sharing] enterprise.”​6 In response, many academic libraries launched consultation services to work with researchers to help them craft data management plans that both met the requirements of funding agencies and were tailored to the needs of the researchers. Librarians have also developed workshops and other educational programming to help researchers better understand what constitutes a strong DMP and to increase their capabilities to produce one.​7 It has been four years since the data management plan requirement was instituted by the NSF. Do researchers now have a better understanding of the data management requirements and how to respond to them? In this paper, the content of twenty­nine DMPs is reviewed and the quality of these DMPs is compared to DMPs from a previous study in 2013. The aims of this research are to identify areas of growth in researchers’ collective understanding of the DMP requirements as evidenced by the DMPs they have written, and to determine particular areas of need to inform how data services offered by the library may be improved. Literature Review Since the advent of the data management plan requirement from NSF, libraries and librarians have been actively exploring their place in the data management plan landscape: “Increasing attention to data management plans can be most readily attributed to their requirement by funders.”​8​ Dietrich et al. found that guidance for the data management plans from funding agencies ranges from being detailed to ambiguous, so there is an opportunity for librarians to help.​9​ Federer writes that “[m]any researchers do not have formal training or expertise in data management, making it difficult for them to meet funder requirements for data sharing.”​10​ While examining an embedded medical informationist’s involvement with a research team, Federer notes that the research team asked the informationist for assistance with writing a DMP.​10​ At Colorado State University, librarians conducted focus groups with faculty and researchers where “a few participants indicated that they would appreciate assistance creating DMPs.”​11​ One participant thought that “all researchers would benefit from experts ‘helping us do it right the first time.’”​11​ In 2012, Diekema et al. surveyed 253 faculty about data services and practices and found that 37.7% of the respondents would like help with writing DMPs.​12 This is a growing area within libraries. Antell et al. surveyed science librarians in 2014 and found that 15.8% of respondents had the job duty described as “help researchers develop data management plans.”​13​ Additionally, 15.1% indicated that they “‘promote, publicize, or advocate’ the library’s data management services.”​13​ When asked what skills are needed, 4.2% of the survey respondents indicated that “experience assisting with data management plans” would be useful, but only 2% thought they had those skills or were working on acquiring them.​13 One method of providing support for researchers is to make DMP templates. Participants in the focus groups at Colorado State University expressed “awareness or prior use of, and appreciation for, the CSU Libraries’ data­management plan templates.”​11​ The University of Illinois Library also provided DMP templates for specific NSF directorates.​14​ Librarians at the University of Houston created an online form for researchers to use when writing a DMP.​15 Similar to a template, DMPTool has been a useful resource that librarians have used and recommended to faculty. DMPTool is an online resource that provides templates and takes users through the creation of DMPs step­by­step. The University of California, Los Angeles spearheaded the development of DMPTool and it “has had significant use among the UC campuses.”​16​ Johnston et al. notes that feedback from the workshop attendees at the University of Minnesota encouraged the libraries “to increase promotion of its free DMP consultation services and highlight its involvement with DMPTool.”​7​ James Madison University also decided to join DMPTool to help their researchers navigate DMP requirements.​17​ A similar resource, DMPOnline, was developed by the Digital Curation Centre (DCC) to support researchers in meeting the requirements of funding agencies primarily in the United Kingdom.​18 Only a couple studies have been published that have evaluated the contents of DMPs. Mischo et al. did an analysis of storage venues and reuse mechanisms mentioned in 1,260 NSF DMPs from all subject areas at the University of Illinois. This study “found that there were no statistically significant differences between the specific storage venues and reuse mechanisms within funded and unfunded proposals.”​14​ Nicholls et al. looked at DMPs from funded NSF proposals from the University of Michigan’s College of Engineering and did an analysis using the NSF Directorate for Engineering’s guidance which included roles and responsibilities, period of data retention, expected data, data formats and dissemination, and data storage and preservation of access.​19 Both of these studies indicate a need for data education for both faculty and graduate students. Consulting with faculty about their data management plans has been a leading initiative in developing data services in academic libraries, but this is by no means the only type of data service being offered or considered. Johns Hopkins University, for example, offers services to support data management and curation across the lifecycle of a data set, from consulting on DMPs to archiving the data using an institutional repository.​20​ Purdue University has also developed an institutional repository that is dedicated solely to disseminating and curating research data​21​ and have engaged in direct partnerships with faculty to address their distinct data needs.​22, 23​ Several libraries are now offering data information literacy programming designed to teach students how to manage and curate the data they produce more effectively.​24, 25​ As Akers reminds us, not all faculty will serve as principal investigators (PIs) on grants that require DMPs, and therefore libraries ought not to focus all of their efforts into data management planning.​26 Nevertheless, analyzing DMPs can be a good starting place in developing an understanding of the data needs of researchers. This understanding can then be applied directly towards improving DMP consultation services as well as identifying and informing other potential data services. Looking forward, more research funders will likely require a DMP as part of a grant application, so writing DMPs will increasingly be part of a researcher’s grant application process. In 2014, the U.S. Department of Energy announced that it would begin requiring DMPs on grant applications.​27​ Halbert wrote about the NSF mandate as “...neither unprecedented nor an isolated intervention by one federal agency...for example, the National Endowment for the Humanities adopted a requirement for data management plans...”​28​ Kozlowski reported on a panel presentation where speakers shared updates on agencies’ responses to the Public Access to Research Results (PARR) memo.​29​ Gherghina and Katsanidou wrote about data management requirements in Europe: “Since 2011, the Economic and Social Research Council, the main funding body of the United Kingdom, requires all funding proposals to include a data management plan...”​30​ Diekema et al. studied the effect of funders mandating data management plans and found that the mandates caused changes in services at some sponsored programs offices, has impacted some researchers’ data management practices, and found that survey respondents felt that “the role of central IT and the library has increased” with regard to helping with data.​12​ As more funders are requiring DMPs, it is clear that the library can provide assistance. Setting This study takes place at the University of Michigan, a large research university with a strong library system. The university consists of nineteen schools and colleges, one of which is the College of Engineering. The College of Engineering has a large and diverse student and faculty body: Tenure­Track and Tenure Faculty (Fall 2013) 381 Research Faculty (Fall 2013) 127 Masters & Doctoral Students (Fall 2014) 3,169 Undergraduate Students (Fall 2014) 6,204 The College of Engineering received over $162 million in federal funding in 2014, of which 20% ($33 million) was from the National Science Foundation. The library at the University of Michigan has been building up support services around data for several years now. Spatial and Numeric Data Librarians were hired beginning in 2005 and have an established space for helping patrons find and use data.​31​ CLIR (Council on Library and Information Resources) Data Fellows were on staff from 2012 ­ 2014 and conducted background research and helped lead the development of internal training for liaison librarians to learn more about data.​31​ In 2014, a Research Data Services Manager was hired to more formally support the increasing data needs of researchers, and to support the liaison librarians whose faculty work with data. In 2013, the authors conducted a rough review of data management plans​19​ that were attached to accepted NSF grant applications to learn: ● more about the DMP requirements ● how well the researchers were addressing those requirements ● what the areas of need were Now that data management plans have been required by the NSF for several years and the researchers have become accustomed to the requirement, it was time for another, more formal, review. Method This more formal review consisted of reviewing twenty­nine DMPs. These DMPs were acquired from the Office of Research in the College of Engineering and were from grant applications that were accepted by NSF between January and June 2014. The DMPs were written by researchers from a wide variety of disciplines within the College of Engineering. The DMPs were evaluated using three different rubrics: the College of Engineering (CoE) Rubric, the University Rubric, and the Data management plan as a Research Tool (DART) Rubric. CoE Rubric The CoE Rubric was the first rubric used to evaluate the first set of DMPs from the College of Engineering. This rubric consists of eight criteria that were established based on the NSF Directorate for Engineering DMP guidance. To apply the CoE Rubric, librarians provided notes under each criterion to illustrate how the DMP met the criterion. If the DMP said nothing about a required criterion, it was coded as “no mention.” In determining overall quality of a DMP, this rubric used a binary system. For each of eight criteria, the DMP was assigned 0 (did not meet, coded as “no mention”) or 1 (met), based on the librarian’s notes. These numbers were totaled, and overall DMP quality was determined on the scale: 0­2 (poor); 3­5 (fair); 6­8 (good). The eight criteria were: Roles and Responsibilities; Types and Formats; Storage; Size of Data; Documentation and Metadata; Dissemination/Provision for Re­use; Archiving and Preservation; and Retention. The CoE Rubric is included as Appendix A. University Rubric The University Rubric was developed by the CLIR Data Fellows for evaluating DMPs that were written by researchers from a different college within the university. The rubric consists of a thorough set of questions that cover all aspects of data collection and management that could potentially be included in a DMP, beyond even the requirements listed by NSF directorates. This rubric was used in addition to the CoE Rubric since it covers requirements beyond the Directorate for Engineering guidance and better facilitates comparisons between DMPs by using a very defined set of outputs. The majority of the criteria use a binary system for indicating if the piece of information is included in the DMP (For example, the responses to “Is the total amount of expected data specified?” are yes/no), but there are some that have more possibilities. For example, “Who will be responsible for data management?” has four potential answers: PI and/or co­PIs, Trainees (graduate students, postdocs or technicians), More than one of the above, and Not clear. The questions asked in the University Rubric are listed in Appendix B. DART Rubric The Data management plan as A Research Tool (DART) project​32​ seeks to evaluate the content of data management plans as a means to inform the development of library services in managing and curating research data. With support from the Institute of Museum and Library Services (IMLS), librarians from Oregon State University, the University of Oregon, Penn State University, the George Institute of Technology and the University of Michigan are developing and testing a rubric for this purpose. In addition to the use of this rubric by individual librarians as a means of determining how closely a researcher has complied with the requirements issued by an NSF directorate, the rubric will enable standardized evaluations of DMPs across multiple institutions. Thus, the library community will have a tool that will produce meaningful comparisons that could lay the groundwork for identifying common issues and creating best practices to address them. This study made use of an early iteration of the DART Rubric and served as a beta test of its effectiveness. The results produced by this study and the experiences of the librarians who made use of the DART Rubric were used to further the development of the rubric. It provided an alternate method of evaluating the DMPs that differed from the CoE Rubric and University Rubric by using a three­tiered method of evaluation. The DART Rubric used in this study consisted of twelve items that matched common DMP requirements mentioned by all NSF directorates. Each item was rated on a three­point scale ­ low, medium, or high ­ depending on the level of detail provided. “Low” means there is no information fulfilling the requirement, “Medium” indicates there is little and/or vague information provided, and “High” indicates that the DMP clearly meets the requirement. The early iteration of the DART Rubric is available as Appendix C. The librarians reviewed the DMPs using all three rubrics, each of which contributed to a more complete understanding of the content of the DMPs. The librarians reviewed DMPs that were written by researchers in their assigned liaison areas ­ two librarians reviewed eleven DMPs each, and the other librarian reviewed seven DMPs. Results One measure of the CoE Rubric, as discussed above, was the overall quality of the DMP. This rubric was originally used in an earlier study of a larger set of DMPs.​19​ The authors of this study were interested in comparing overall quality of DMPs between the two studies to determine if there was any change in quality since the first study. Findings are shown below. N/A refers to documents that stated a DMP was not needed due to the nature of the project. Looking at the overall quality of the DMPs, the DMPs reviewed for this study were slightly better than the DMPs reviewed in 2013. Although only an additional 2% of DMPs in this study were considered “good” or “fair,” there was a marked increase in “good” DMPs, from 19% in the original analysis to 34% in this analysis. Only one DMP reviewed during this current analysis rated as high as it could on the CoE Rubric, scoring an 8 on the 0­8 scale. Although the DART Rubric did not include an overall quality score, information can be inferred by tallying the number of “High” and “Low” marks for each DMP. Only two DMPs had zero “Low” scores across all twelve areas, and only three more had just one “Low” score. None of the DMPs were rated “High” across all twelve categories. The University Rubric did not include an overall quality score. Roles & Responsibilities One of the first items looked for in the CoE Rubric and University Rubric was whether a specific person was listed as being responsible for the data produced by the research project. As expected, when a responsible person was mentioned it was usually the PI or co­PI, but it was also interesting that in 45% of the DMPs, it was not made clear who was in charge of the data.